SW cost estimation: Measuring model performance of arbitrary function approximators

Estimating software development cost with high accuracy is still a largely unsolved problem. Consequently, there is ongoing, high activity in this research field; a large number of different estimation models ranging from mathematical functions to arbitrary function approximators (AFA’s) have been proposed over the last 20+ years. Unfortunately, the studies do not converge with respect to the question “which model is best?” when functions and AFA’s are compared. So far, it has not been understood why this is so. In this empirical study, we show that this is due to inappropriate validation methods as far as the validation of AFA’s is concerned. In fact, the de facto validation method, crossvalidation combined with MMRE, will give completely arbitrary results for AFA’s. Obviously, other criteria are called for in order to appropriately assess the performance of AFA’s. This should be a topic of future research.